An Automated Method for Studying Interactive Systems

Information Retrieval experiments rarely examine more than a small number of user or system characteristics because of the limited availability of human subjects. In this article we present an interaction model, and based on that an experimental method. This automated method helps identify which user and system variables are relevant, which are independent of other variables, and which ranges of the variables are most important. In an automated method there are no human subjects: the human's role is performed by a computer program. This restricts the user variables that can be studied to those whose effect on behaviour can be accurately modeled. To illustrate the automated method, the following experiment will be presented. The subject's task is to hunt down a set of ten similar documents by providing relevance feedback and using query refinement. Performance is measured by precision in the top 10 documents at the end of a trial. Six user and system variables are varied. The experiment runs 600 different combinations of variable values on 50 tasks. This highlights one of the main advantages of an automated study it can be much larger scale than is normally possible with human subjects. The results of this automated experiment show, as human studies have done previously, that relevance feedback consistently improves retrieval performance. Further, they show that in the context of the present task, the effect saturates after the second iteration. Irrelevant and interacting variables are identified. These observations give specific guidance about designing a follow-up study using human subjects.